Journal of Research in Rural Education, 2015, 50(4) 


Identifying, Analyzing, and Communicating Rural: 
A Quantitative Perspective 


Natalie A. Koziol 
Ann M. Arthur 
Leslie R. Hawley 
James A. Bovaird 
Kirstie L. Bash 
Carina McCormick 
Greg W. Welch 
University of Nebraska-Lincoln 
National Center for Research on Rural Education (R 2 Ed) 

The Nebraska Center for Research on Children, Youth, Families & Schools (CYFS) 


Citation: Koziol, N. A., Arthur, A. M., Hawley, L. R., Bovaird, J. A., Bash, K. L., McCormick, C., & 
Welch, G. W. (2015). Identifying, analyzing, and communicating rural: A quantitative perspective. 
Journal of Research in Rural Education, 50(4), 1-14. 


Defining rural is a critical task for rural education researchers, as it has implications for all phases of a study. However, it is 
also a difficult task due to the many ways in which rural can he theoretically, conceptually, and empirically operationalized. 
This article provides researchers with specific guidance on important theoretical and operational considerations relevant 
to conducting quantitative rural education research: identifying a rural definition, selecting appropriate analytic methods, 
and thoroughly communicating rural details to situate the findings within the broader rural literature base. In addition, this 
article uses the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) and three rural definitions 
to illustrate how parameter estimates and substantive interpretations are impacted by the statistical model, rural definition, 
and exclusion/inclusion of covariates. We believe that informed consideration and implementation of the article s guidelines 
will enhance and clarify the quantitative literature on rural education. 


“Rural” is a theoretical construct, so identifying 
a theoretical perspective of rural is a critical first step in 
conducting research on rural education. However, an equally 
critical (and closely related) step is identifying an operational 
definition of rural, which is necessary for conducting 
quantitative rural research. Numerous theoretical and 
operational definitions have been proposed across a wide 
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array of disciplines, and each of these definitions has its 
own strengths and weaknesses. Although the challenges of 
defining rural are well-documented (e.g., Coladarci, 2007; 
Cromartie & Bucholtz, 2008; Hart, Larson, & Lishner, 
2005; Howley, Theobald, & Howley, 2005; Isserman, 2005), 
discussions have primarily occurred at a theoretical level or 
do not delve into the issues that arise once a definition has 
been chosen. Concrete examples and guidelines are needed 
to ensure that researchers fully understand the extent to 
which the rural definition impacts the study’s sampling 
design, analysis plan, and generalizability. 

The purpose of this article is twofold. First, we aim 
to remind (or potentially inform) rural researchers how to 
(a) identify an operational definition of rural given their 
theoretical perspective and the context and goals of their 
study, (b) appropriately analyze their data given the chosen 
operational definition, and (c) accurately communicate 
their findings given the chosen operational definition. Upon 
examining several quantitative articles recently published 
in JRRE, we were generally encouraged by JRRE authors’ 
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careful attention to defining and discussing rural at both the 
theoretical and operational levels. Our goal is to explicate 
and thus foster and sustain this good practice by providing 
a comprehensive guide for education researchers seeking 
to identify, analyze, and communicate rural phenomena. 
Second, we use data from the Early Childhood Longitudinal 
Study, Kindergarten Class of 1998-99 (ECLS-K; developed 
by the U.S. Department of Education, Institute of Education 
Sciences, National Center for Education Statistics [NCES], 
n.d.-a) to illustrate the impact of the rural definition on 
statistical results and substantive inferences. 

It is important to acknowledge that our discussion is 
primarily intended for researchers conducting quantitative 
research. Some of our recommendations may not apply to, 
or may even counter what is recommended for, qualitative 
research. We believe that both forms of research, in addition 
to mixed-methods research, are important for understanding 
rural issues. Thus, we strongly encourage other researchers 
to advise the field on rural definition issues in the context of 
qualitative and mixed-methods research. 

Identifying, Analyzing, and Communicating Rural 
Identifying an Appropriate Definition 

Choosing a rural definition influences the entire scope 
of a study. At the initial planning stages of a study, the 
definition affects the selection of a sampling design and 
statistical analysis plan. At the concluding stages of a study, 
the definition affects the generalizability of the research 
findings. In this section we provide guidance on identifying 
an appropriate operational definition of rural given a 
particular theoretical perspective, while taking into account 
practical considerations. 

Theoretical perspectives. First and foremost, 
operationalizing rural requires formulating a theoretical 
perspective of rural. Numerous theories of rural have 
been postulated. In following the organization scheme of 
Brown and Schafft (2011), such theories can be broadly 
classified into one of two groups: place-based theories (e.g., 
demographic, population, spatial, political economic, and 
socio-cultural theories) and social constructivist theories. 1 
We briefly touch on each of these theories below. A full 
discussion is beyond the scope of this article, so we strongly 
encourage researchers to consult the original sources and 
seek out additional references for a deeper and more 
comprehensive understanding of rural theory. 

One means for conceptualizing rural is in terms of 
“population and settlement structure and landscape” 
(Brown & Schafft, 2011, p. 5), where emphasis is placed on 

'We are particularly grateful for the feedback of an 
anonymous reviewer who outlined and summarized the 
theories discussed in this section. 


demographic characteristics such as population density and 
size, and spatial delimiters. Demographic and population 
density-based definitions of rural largely date back to 
Emile Durkheim’s 1893 work, The Division of Labor in 
Society. Durkheim differentiated societies by the nature of 
their solidarity, which he theorized to be a direct function 
of population density (Durkheim, 1964). According to 
Durkheim, low population density societies (i.e., rural 
societies) lend themselves to a mechanical form of solidarity 
characterized by collectivist orientations, homogeneous 
backgrounds and belief systems, and agrarian lifestyles. 
In contrast, high population density societies (i.e., urban 
societies) increasingly demonstrate an organic form of 
solidarity characterized by an interdependency among others 
that stems from the division of labor. Along with population 
characteristics such as size and density, rural places are 
often spatially defined. Spatial conceptualizations of rural 
focus on the where (i.e., space, distance, and relationship to 
the city) of places (Lobao, Hooks, & Tickamyer, 2007) and 
highlight issues of spatial inequality or spatial disparities in 
the allocation of resources. 

Most operational definitions of rural used in quantitative 
research are grounded in the demographic and spatially 
based theories, as these theories provide a straightforward 
means for classifying geographies. However, just because a 
theory readily enables classification (which in turn enables 
quantification) does not mean it is superior. Other place- 
based theories that emphasize political-economic and socio¬ 
cultural distinctions have also garnered attention. With 
respect to these alternative theories, Brown and Schafft 
(2011) note that although “the focus is not on the space 
(or place) itself ... locales often create a powerful context 
for collective identity and social interaction” (p. 39) such 
that demographic and spatially based classifications can 
sometimes serve as proxies for getting at these alternative 
(and generally harder to operationalize) ways of thinking 
about rural. 

As the name suggests, political-economic theories of 
rural focus on distinctions among sectors that are primarily 
politically and economically driven (Cloke, 2006). Such 
distinctions include the tendency for rural economies to 
be more specialized (Deavers, 1992) and more dependent 
on the government sector (Deavers & Brown, 1985). 
Tilly (1974) attributes this dependence to state-making, 
urbanization, industrialization, and commercialization. 
Much of the literature on class relations (e.g., Stinchcombe, 
1961), including peasant studies (e.g., Wolf, 1969), can 
also be considered here. While conceptually distinct, many 
of these issues relate, at least indirectly, to Durkheim’s 
(1964) theorizing on rural, suggesting that demographic 
characteristics could be used in some instances to 
approximate political economic classifications. 

Socio-cultural theories frame rural in terms of formal 
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and informal social and cultural networks (e.g., Schafft, 
2000) and interactive community fields (e.g., Kaufman, 
1959; Wilkinson, 1972). Schafft and Brown (2000) note that 
“embedded intra-community relations, including individual 
and group-level social ties, cultural practices, and political 
behavior, reinforce the affiliative networks within a given 
locality” (p. 204). Strong ties among members within a 
community promote social cohesion, while ties (albeit 
weaker) with wider networks (e.g., ties between rural 
communities and neighboring cities) ensure sufficiency of 
resources and prevent isolation (Schafft, 2000). Although 
these networks are socially defined, spatial influences are 
clearly evident. With respect to interactional communities, 
Kaufman (1959) explains that “at the cultural level, 
integration is effected through the widely shared values 
and objectives pertaining to the community field and, at the 
ecological level, through a ‘functional relation’ of services” 
(p. 12). Discussions of social norms (e.g., reciprocity 
and civic engagement) as they relate to social capital 
(e.g., Putnam, 1993) also contribute to social-cultural 
conceptualizations of rural. 

The issues discussed in the remainder of this article 
relate most directly to place-based theories of rural, 
particularly the demographic and spatially based theories 
of rural. However, we would be remiss not to mention 
social constructivist approaches to conceptualizing rural. 
In particular, Halfacree (1993, 2006) emphasizes the 
importance of considering non-tangible indicators of space 
such as cognitive structures and social representations. 
Place identity construction, specifically the notion of urban 
and rural identities (Bell, 1992; Creed & Ching, 1997), falls 
under this approach. Bell (1992) observes that, although 
“the difference between country life and city life may only 
ever be true in the mind” (p. 66), perceived differences have 
real social consequences. 

Operational definitions. To conduct quantitative 
research, one’s theoretical perspective of rural must be 
translated into operational terms. A reasonable first step 
in searching for an appropriate operational definition is 
to examine the definitions that are currently in use. Most 
well-established and commonly used definitions have been 
developed by one of four federal agencies/centers. The U.S. 
Department of Agriculture, Economic Research Service 
(ERS) has developed the Rural-Urban Continuum Codes 
(RUCCs, also referred to as the Beale Codes; ERS, 2013a), 
Urban Influence Codes (UICs; ERS, 2013b), and Rural- 
Urban Commuting Areas (RUCAs; ERS, 2013c), developed 
jointly with the Washington, Wyoming, Alaska, Montana, 
and Idaho Rural Health Research Center (WWAMI RHRC, 
n.d.-a). The U.S. Office of Management and Budget (OMB) 
has defined Core Based Statistical Areas (CBSAs), which 
include Metropolitan and Micropolitan Statistical Areas 
(OMB, 2010). In addition, the U.S. Census Bureau (2013a) 


has created an Urban and Rural Classification. Finally, 
NCES has developed the Urban-Centric Locale Codes 
(NCES, n.d.-b), a revised version of the previously used 
Metro-Centric Locale Codes (or simply, the Locale Codes; 
NCES, n.d.-c). Detailed descriptions and comparisons of 
the definitions’ strengths and weaknesses are provided by 
Arnold, Biscoe, Farmer, Robertson, and Shapley (2007); 
Coburn et al. (2007); Hart (2012); and Hart et al. (2005). 

Given the prominence and usage of these ways of 
classifying geographies, we strongly encourage researchers 
to consider the definitions seriously. At the same time, we 
recognize that the definitions have limitations and may not 
be appropriate for all study contexts. As an alternative, a 
number of researchers have created their own rural definition 
by modifying or combining one or more of the existing 
definitions. For instance, Isserman (2005) introduced a 
Rural-Urban Density Typology that combines elements of 
the U.S. Census Bureau and OMB definitions—including 
county population density, percentage of county population 
in urban/rural areas, and presence/absence of urban areas 
of 10,000+ or 50,000+—to acknowledge the presence 
of “mixture” counties that contain both rural and urban 
areas. As another example, Waldorf (2006) proposed the 
Index of Relative Rurality, a continuous measure of rural 
comprised of population size and density, extent of urban 
area, and remoteness. A primary advantage of the index is 
that it avoids the need to impose arbitrary thresholds such as 
10,000+ and 50,000+. 

Primary considerations. In creating a new definition 
or choosing among well-established definitions, researchers 
must ultimately bear in mind their theoretical perspective of 
rural. Different definitions of rural use different operational 
indicators; the most relevant indicators are those that match 
the researcher’s theoretical perspective. For example, the 
U.S. Census Bureau (2013a) uses a definition of urban 
and rural that is based primarily on population size and 
density. This definition clearly links to demographic-based 
theories of rural. A moderately strong relationship has been 
identified between population density and human capital 
investment (Garces-Voisenat, 2011), suggesting that in 
education research population density might be important 
for explaining job growth for teachers or other school 
officials. The OMB and RUCA classification systems 
additionally take commuting patterns into account (OMB, 
2010; WWAMI RHRC, n.d.-b) and thus relate to spatially 
based theories of rural. Commuting patterns might be 
particularly relevant to education researchers who want 
to understand the impact of access to big-city resources, 
such as art museums or institutions of higher education, on 
child outcomes. Similarly, the Urban-Centric Locale Codes 
stem from spatial conceptualizations of rural, as they rely 
on geographic information systems (GIS) technology to 
classify locations (NCES, n.d.-d). Researchers might use 
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this definition to study the impact of geographic remoteness 
on schools’ reliance on distance technology. 

Consistent with the guidelines of Hart et al. (2005), the 
indicators described above are quantifiable and relatively 
objective. However, as noted by Coladarci (2007), 
some researchers question the meaningfulness of such 
“traditional constructs of demography” and argue for “more 
important notions of ‘local commitments’ and ‘meaning¬ 
making’” (p. 2). In particular, none of the aforementioned 
definitions directly captures the tenets of the political- 
economic, socio-cultural, and social constructivist theories. 
If researchers deem demographic and spatial characteristics 
to be insufficient proxies for operationalizing rural under 
these alternative theories, then other indicators should be 
explored. In doing so, however, researchers must be willing 
and able to provide convincing evidence that the potentially 
subjective indicators do in fact measure what they are 
purported to measure. 

Related to choosing a rural indicator) s) is determining the 
geographic unit to which the indicator)s) should be applied. 
This consideration is important because the geographic 
unit represents the experimental unit—the smallest unit to 
which the “treatment” is independently applied (Milliken & 
Johnson, 2009)—for the rural/urban predictor. As discussed 
in the “Analyzing Rural Data” section, inferences about 
rural exist at the level of the experimental unit (e.g., the 
county), which is not always the same as the lowest-level 
sampling unit (i.e., the lowest-level unit for which data are 
being collected, for example, the student). 

Common geographic units include schools, school 
districts, ZIP code areas, census tracts, and counties. 
Obviously, researchers should choose a geographic unit that 
matches their target sampling unit. For example, the school- 
level definitions provided by the Metro- and Urban-Centric 
Locale Codes might be particularly useful for comparing an 
intervention’s efficacy in rural versus urban schools. On the 
other hand, county- level definitions provided by the RUCCs 
and UICs would be more appropriate for evaluating rural 
and urban county health and wellness initiatives. County- 
level definitions may also be advantageous in the context 
of multi-year longitudinal studies, as counties tend to be 
more stable over time compared to other geographical 
units (Hart et al., 2005). When the ideal geographic unit 
is not immediately apparent, researchers may benefit from 
choosing the geographic unit that has the most variation on 
the outcome variable. For instance, suppose “professional 
development opportunities” is the outcome variable. If most 
of the variability in professional development opportunities 
exists between school districts as opposed to within school 
districts, then school-district-level predictors will generally 
account for more variability in professional development 
opportunities than, say, school-level predictors. Hence, in 
this context, rural definitions applied at the school-district 


level are likely preferred over rural definitions applied at 
the school level. 

The choice of geographic unit strongly influences 
conclusions about rural phenomena, a consequence that has 
been defined in the literature as the modifiable areal unit 
problem (MAUP). Waller and Gotway (2004) describe 
MAUP as the “geographic manifestation of the ecological 
fallacy in which conclusions based on data aggregated to 
a particular set of districts may change if one aggregates 
the same underlying data to a different set of districts” (p. 
104). MAUP involves both a scale/aggregation problem 
and a grouping/zoning problem. The scale/aggregation 
problem relates to the fact that statistical results will vary 
as a function of the level of aggregation applied. As Arnold 
et al. (2007) note, greater aggregation generally results in a 
greater loss of information—e.g., many counties classified as 
metropolitan contain significant rural spaces and vice versa 
(Isserman, 2005). The grouping/zoning problem relates to 
the fact that statistical results will vary depending on how 
groups are formed, given a particular level of aggregation. 
For example, census tract boundaries often change across 
census years, resulting in different groupings across census 
years (Hart et al., 2005). While there is no definitive solution 
to MAUP, researchers should be aware of the implications 
of choosing a particular geographic unit, and shape their 
inferences about rural phenomena accordingly. 

Supplemental considerations. Sometimes logistical 
concerns make a theoretically ideal rural operationalization 
practically infeasible. One such concern is financial 
constraints. Consider a definition that is applied to counties. 
The power to detect a rural effect is primarily influenced 
by the number of counties sampled. Sampling counties is 
likely to be much more expensive than sampling schools, 
which tend to exist in greater numbers and in narrower 
geographical regions. Researchers may not have the time 
and resources to sample at the county level. Likewise, 
choosing one of the broader definitions of rural (e.g., the 
U.S. Census Bureau’s definition) will result in more places 
qualifying as in-need based on policy or program eligibility 
requirements, but programs may be limited in their funding 
(Coburn et al., 2007). 

Another concern is the availability of data. Coburn 
et al. (2007) offer a health policy example in which they 
note that provider claims are based on ZIP codes. Without 
additional information, rural definitions applied to counties 
cannot be used to address questions about provider claims, 
because counties and ZIP codes are not directly comparable. 
ZIP codes were created by the U.S. Postal Service as mail 
delivery areas and are not bound to counties or even states 
(“Can ZIP codes cross,” 2011) and can change monthly 
(“How many changes are made,” 2011). Although ZIP 
codes may be loosely aggregated with GIS techniques 
into approximate counties (DuScheid, 2011), this practice 


IDENTIFYING, ANALYZING, AND COMMUNICATING RURAL 


5 


introduces additional error as it does not always provide an 
exact match. In the context of a secondary data analysis, 
researchers are limited to the data at hand. If the dataset 
does not provide NCES school IDs or census tract or county 
codes (either the American National Standards Institute 
[ANSI] codes or the older Federal Information Processing 
Series [FIPS] codes; U.S. Census Bureau, 2013b), then the 
corresponding definition codes cannot be merged with the 
dataset. 

Finally, a dataset may include all the necessary 
identification and coding information, but whether it is 
appropriate to make comparisons based on higher-level 
geographic units depends on the sampling design of the 
original study. For instance, although the restricted-license 
ECLS-K dataset provides information on higher-level units, 
the study’s website states that “the ECLS-K sample was not 
designed to support state-level (or city- or county-level) 
estimates, as the sample is not necessarily representative of 
children in particular states (or cities or counties)” (NCES, 
n.d.-e). 

To conclude this section, we stress the hierarchy of 
theoretical and practical considerations. Ultimately, theory 
should be the driving force in selecting an operational 
definition of rural. Practical considerations, while important, 
should serve more as an evaluation of the feasibility of the 
chosen rural definition, and potentially as an indication of 
the need for additional or alternative study resources. 

Analyzing Rural Data 

Once a rural definition is chosen, the next step is to 
determine an appropriate analytic strategy. The analysis 
plan follows directly from the sampling design, which in 
large part depends on the geographic unit to which the rural 
definition is applied. In education and rural research, simple 
random samples are generally impractical and inefficient. 
Instead, researchers often use complex sampling procedures 
that involve clustering (also referred to as nesting). Clustering 
is a sampling method in which the units being sampled 
contain multiple related observation units (Kish, 1965). 
For instance, rather than directly sampling students, it may 
be more convenient to randomly sample schools and then 
observe a selection of students within each school. Whereas 
clustering is often more time and cost efficient than simple 
random sampling (Kish, 1965), a disadvantage of clustering 
is that the assumption of independent observations, implicit 
to most basic statistical models and tests, is violated. 

Consider the following scenario. Suppose a researcher 
is interested in comparing the body mass index (BMI) of 
rural and urban children, where the terms rural and urban are 
defined at the school level. The researcher randomly selects 
10 rural schools and 10 urban schools, and within each 
school, measures the BMI of 10 randomly selected children. 


The sampling design involves two levels where level 1 units 
(the lowest-level sampling units; = 200 children) are nested 
within level 2 units (the experimental units for the rural/ 
urban predictor; = 20 schools). At first glance, it might seem 
that an independent samples t-test comparing the average 
BMI of the 100 children in rural schools to the average 
BMI of the 100 children in urban schools is sufficient for 
testing the “rural” effect on BMI. However, the assumption 
of independent error terms is violated. Children who go to 
the same school will tend to be more similar than children 
who go to different schools, so in actuality, there are 10 
independent observations per group instead of 100. Only 
considering data at the lowest level of sampling is referred to 
as disaggregation (Snijders & Bosker, 2012). In the simplest 
case, when predictor variables are assigned at a higher level 
of sampling, disaggregation increases the chance of a Type 
I error (e.g., finding urban and rural differences when there 
are no differences). In contrast, when nesting is present (and 
ignored) but predictor variables are measured at the lowest 
level of sampling, disaggregation increases the chance of a 
Type II error (e.g., failing to find urban and rural differences 
when there are in fact differences). 

An alternative is to aggregate the lower-level data to 
the higher level. This approach would involve computing 
the average BMI for each school and then performing 
an independent samples t-test to compare the average 
BMI of the 10 rural schools to the average BMI of the 10 
urban schools. Like disaggregation, aggregation has its 
disadvantages. Aggregation throws away within-cluster 
(i.e., within-school) variability, which prevents researchers 
from fully understanding all of the variability in the outcome. 
For instance, research has shown that the prevalence of 
obesity among children is greater for low-income families 
(Ogden, Lamb, Carroll. & Flegal, 2010), suggesting that 
socioeconomic status (SES) is an important predictor to 
include in any analysis of BMI. In the case of an aggregated 
analysis of BMI, the only means for including SES as a 
predictor would be to aggregate family-level SES to the 
school level and then regress average BMI on average SES 
and rural status. Importantly, family-level SES and school- 
level SES are distinct predictors with potentially distinct 
effects on BMI. The effect of school-level SES on BMI may 
be weaker, stronger, or even in the opposite direction 2 of 
the effect of family-level SES. Drawing inferences about 
lower-level relationships based on higher-level relationships 
is referred to as an ecological fallacy (Snijders & Bosker, 
2012). In an attempt to avoid such a fallacy a researcher 
might perform two separate regressions, one at the child 
level and one at the school level. This approach is not ideal. 
First, all else being equal, the estimators of the regression 

2 For instance, we might find this case if a county 
initiative aggressively targeted low SES schools for healthy 
lunch programs. 
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coefficients will be less statistically efficient (resulting in 
greater standard errors) when the analyses are conducted 
separately. Second, separate analyses preclude the 
possibility of cross-level interactions (e.g., the interaction 
between child-level SES and school-level rural status). This 
limitation is serious. Rural education research is complex, 
and examining interactions is one way to acknowledge the 
complexity (Howley et al., 2005). 

Aggregation and disaggregation leave much to be 
desired. An alternative approach to handling clustering 
is multilevel modeling (MLM), 3 also referred to as linear 
mixed modeling, hierarchical linear modeling (not to be 
confused with hierarchical multiple regression), and random 
coefficients modeling (Raudenbush & Bryk, 2002). MLM 
avoids the limitations of aggregation and disaggregation by 
partitioning variance in the outcome into its multiple sources 
and then modeling these sources simultaneously. It is useful 
(albeit an oversimplification) to conceptualize MLMs 
simply as regression models with more than one error term. 
As an example, the unconditional (i.e., no predictor) model 
for the two-level BMI scenario can be expressed in equation 
form as 

Level 1: v. = B . + e.. (1) 

Level 2: R . = y + u . (2) 

which can be simplified by replacing the P 0 placeholder in 
Equation 1 with Equation 2: 

Combined: v.. = y + u .+ e ■ (3) 

3 Another option for handling clustered data is to 
calculate empirical standard errors that adjust for the 
complex sampling design (e.g., Wolter, 2007). 


In Equation 3, y. represents the BMI of the ; lh child who 
attends the / h school, y 00 is the grand mean, u is the 
difference between the f h school’s mean and the grand 
mean, and e is the difference between the i' h child’s BMI 

'J 

and the / h school’s mean. The only difference between a 
traditional regression model and Equation 3 is the presence 
of the additional school-level error term (u ). Figure 1 
provides a small illustration of each of the terms in Equation 
3 based on hypothetical data for two schools (j= 1,2), each 
with two children (1=1,2). We see that the mean BMI for 
School 1 (y = 25) is lower than the grand mean (y 00 =26), 
so u m is negative. In contrast, Child 1 from School 1 has 
a higher BMI (y =25.54) than the mean BMI for his or her 
school, so e n is positive. If we did not include the school- 
level error terms, the residual would instead represent the 
distance between the child’s BMI and the grand mean, 
which would result in correlated error terms due to the 
fact that children from the same school tend to have more 
similar BMIs than children from different schools (in our 
hypothetical example). 

Earlier we noted that MLM serves to partition variance 
in the outcome into its between- and within-level sources. 
The proportion of variability that exists at the higher level 
can be estimated via an intraclass correlation coefficient 
(ICC) 


where erj is the between-level (e.g., between-school) 
variance in the outcome, and CT W is the within-level (e.g., 
within-school or between-child) variance in the outcome. 
The larger the ICC, the greater the proportion of variability 
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Figure 1. An illustration of the regression terms presented in Equation 3. 
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that exists at the higher level. By calculating the proportion 
of variability in the outcome that exists at each level we can 
determine which types of predictors will be most helpful 
in explaining the outcome. If very little variability in BMI 
exists at the school level then a rural definition applied at 
the school level will probably not tell us much about BMI. 

We conclude this discussion of MLM with a 
demonstration to illustrate the consequences of ignoring 
clustering and omitting important covariates. Our review of 
the quantitative articles published in JRRE between 2009 
and 2013 revealed only one study (Stockard, 2011) that used 
MLM. 4 This finding by no means indicates that the studies 
that did not use MLM should have used MLM (MLM is only 
applicable under certain conditions), but it suggests that a 
demonstration might be worthwhile. The interested reader 
should see Durham and Smith (2006), Reeves and Bylund 
(2005), and Roscigno and Crowley (2001) for additional 
examples of MLM in rural education research. Readers who 
are new to MLM and interested in learning more about the 
topic should see one of the many comprehensive textbooks 
on MLM methods (e.g., Raudenbush & Bryk, 2002; Singer 
& Willett, 2003; Snijders & Bosker, 2012). 

Drawing on our hypothetical example, we simulated 
BMI data for 200 children from a total of 20 schools (10 
“rural” and 10 “urban”). To use MLM, the data must be in 
stacked (also referred to as person-period or long) format— 
that is, each row should correspond to the lowest-level unit 
(i.e., the child, in our example). The datafile and model 
syntax (both SAS and SPSS syntax) are available from the 
first author upon request. 

As a first step we calculated the ICC by analyzing the 
unconditional two-level model (Model 1) corresponding to 
Equation 3. Table 1 provides partial results. Approximately 
(2.77/[2.01 + 2.77]) * 100% = 58% of the variability in 
BMI is at the between-school level. This bodes well for our 
school-level rural predictor. 

As an example of disaggregation, we evaluated the 
effect of rural status on BMI using a single-level model that 
ignored the existence of school-level variance (Model 2). 
Based on 198 degrees of freedom, we would determine that 
rural status has a significant effect on BMI (p < .001). Of 
course, this conclusion is untrustworthy because the model’s 
assumption of independent error terms is violated. Next we 
evaluated Model 3, a multilevel model that accounted for 
the nesting of children within schools. Table 1 shows that 
the estimated beta coefficient representing the effect of 
school rural status is the same for Models 2 and 3 (note 
that this result may not hold perfectly in more complex 
data situations). However, for Model 3, the test of the rural 

4 Irvin, Farmer, Leung, Thompson, and Hutchins (2010) 
discuss MLM but do not actually employ the method. 


Table 1 


Single-Level and Multilevel Model Estimates for Simulated 
BMI Data 





Yoi 

SE 

df 

P 

Model 1 

2.01 

2.77 

— 

— 

— 

— 

Model 2 

3.20 

— 

2.42 

0.25 

198 

<.001 

Model 3 

2.01 

1.31 

2.42 

0.55 

18 

<.001 

Model 4 

0.90 

0.90 

0.24 

1.06 

16.98 

.827 


Note. a Denominator degrees of freedom were estimated using a 
Satterthwaite approximation, dy, = within-cluster variance 
(residual variance), dj = between-cluster variance, foi = effect 
of rural on BMI. Model 1 = unconditional two-level model. 

Model 2 = single-level model with rural predictor. Model 3 = 
two-level model with rural predictor. Model 4 = Model 3 + 
child-level and school-level SES predictors. 

effect is only based on 18 degrees of freedom resulting in 
a larger standard error. In examining the estimated variance 
components for Model 3, it is not a coincidence that the 
within-level variance is the exact same as the within- 
level variance estimated for Model 1, but the between- 
level variance is reduced (again, this result may not hold 
perfectly in more complex data situations). Rural status is 
the same for all children within a particular school so it 
cannot explain within-school variance. 

A key consideration when analyzing rural data is 
whether covariates should be included in the model. As 
Coladarci (2007) emphasizes, “without adequate controls 
in place, the obtained [rural/urban] differences may be 
either unwittingly exaggerated or understated (although 
exaggeration is more likely)” (p. 4). Earlier in this section 
we mentioned the relationship between SES and BMI. Table 
1 indicates that, upon controlling for child- and school- 
level SES (Model 4), we would conclude that school rural 
status is not a significant predictor of BMI (p = .827). What 
appeared to be a rural phenomenon based on Models 2 and 
3 was actually explained by SES. 

Our discussion and examples of MLM focused on only 
one possible source of dependency among observations, 
dependency due to nesting or clustering of lower-level 
units within higher-level units. Dependency can occur for 
a multitude of other reasons. Particularly relevant to rural 
education research is the possibility of geographic, or 
spatial, dependency (see Kucerova & Kucera, 2012, for a 






KOZIOL, ARTHUR, HAWLEY, BOVA1RD, BASH, MCCORMICK & WELCH 


discussion of “the geographical aspects of education,” p. 
3). Goodchild (1992) defines spatial dependence as “the 
propensity for nearby locations to influence each other 
and to possess similar attributes” (p. 33). For instance, it 
certainly seems plausible that counties in the Northeastern 
region of the United States are more similar to one another 
than to counties in the Southeastern region of the United 
States. A number of methods have been developed for 
analyzing spatially dependent data (see, for example, the 
two-level spatial model described by Verbitsky-Savitz & 
Raudenbush, 2009). In fact, an entire branch of statistics 
is devoted to the study and analysis of spatial dependency. 
While an in-depth discussion of spatial methods exceeds 
the scope of this article, interested readers should seek 
out additional information (e.g., Cressie, 1993; Gelfand, 
Diggle, Fuentes, & Guttorp, 2010). 

The purpose of this section was to remind readers about 
some of the critical issues that arise when analyzing rural 
education data. Although educational outcomes are most 
often measured at the level of individual children, teachers, 
principals, etc., rural definitions are most often applied at 
higher levels such as census tracts or counties. Dependency 
among observations must be taken into account when 
considering the effect of rural on the outcome. In addition, 
alternative explanations of rural findings should be evaluated 
through the inclusion of covariates. 

Communicating Rural Findings 

A study is not complete until results have been 
disseminated. Providing transparent and detailed accounts 
should be a goal of all researchers, but it is perhaps even 
more critical for rural education researchers given that 
many theoretical perspectives and operational definitions of 
rural exist. Using the nondescript label of rural is to commit 
a nominal fallacy; rural alone does not actually explain what 
is being measured. 

Appropriate communication of rural findings starts 
with the literature review. In describing previous rural 
studies, it is important to note how researchers defined (or 
did not define) rural. For example, in reviewing the findings 
of Farmer et al. (2006), Irvin, Farmer, Leung, Thompson, 
and Hutchins (2010) condition their discussion based on 
the operational definition that was used: “African American 
youth attending schools identified as Rural Low Income by 
the Rural Education Achievement Program are four times 
less likely to meet Adequate Yearly Progress” (p. 2). The 
goal is to appropriately situate the present study within 
the broader rural research context rather than limit the 
discussion to studies that use a similar definition. In fact, the 
presence of different operational definitions adds richness 
to the field. Operational definitions are merely proxies for 
theoretical definitions of rural, so determining whether 


a particular finding holds across different operational 
definitions is noteworthy (e.g., by exploring alternative 
ways of combining the Beale Codes and observing no 
significant impact on the results, Jordan, Kostandini, and 
Mykerezi [2012] were able to have more confidence in the 
robustness of their results). 

As with any method section, researchers should provide 
readers with enough detail to replicate the study. Limiting 
the description to generic terms such as “rural” and “urban” 
does not facilitate replication. At a minimum, it is necessary 
to specify the indicators used in defining rural and the 
unit to which the definition was applied. Along with this 
description, researchers should provide a strong rationale 
for why they chose the definition. A nice example of such a 
rationale is given by Jordan et al. (2012): 

Beale Codes were used here because they were 
designed specifically to examine the continuum 
between urban and rural areas. They were 
developed for the analysis of trends in non-metro 
areas that are related to the population density 
and metropolitan influence. Beale Codes allow a 
more detailed analysis of the survey data than the 
more common urban-suburban-rural classification 
systems (p. 4). 

Not only does this explanation inform the reader, it forces 
the researchers to carefully consider whether the definition 
is indeed appropriate given their theoretical perspective and 
the context of their study. 

Appropriate communication concludes with a 
discussion section that is framed by the researcher’s 
theoretical perspective of rural but couched in terms of the 
operational definition that was applied. Again, operational 
definitions are merely proxies, so researchers should avoid 
making unjustified generalizations. Hannum, Irvin, Banks, 
and Farmer (2009) provide a great example of limiting their 
conclusions to the population from which they sampled. 
They emphasized that “the results are generalizable to rural 
schools meeting the definitions in the REAP program. The 
results may or may notbe the same in urban, suburban, orrural 
schools other than those identified in the REAP definition” 
(p. 13). As recognized by Hannum and colleagues, it is 
important to limit generalizations to the geographic unit to 
which the definition was applied. For example, if the rural 
definition was applied at the county level then the effect 
of rural should be discussed at the county level. A higher- 
level predictor cannot explain lower-level variation in the 
outcome. That is, a county-level rural predictor cannot 
explain why individuals within counties perform differently 
on the outcome of interest; rather, it can only explain why 
counties perform differently on the outcome of interest. 
Likewise, it is more revealing and precise to relate findings 
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to the observable indicators used to operationally define 
rural (e.g., “remoteness alone did not compromise access 
to technology” [Howley, Wood, & Hough, 2011, p. 6]) 
rather than use the elusive terms “urban” and “rural.” This 
practice is critical for public policy and the development of 
educational programs that rely on tangible indicators. 

As we demonstrate in the next section, different 
operational definitions of rural can lead to very different 
results. This consequence does not mean that certain 
definitions are inherently wrong, but it does mean that 
choosing a definition requires careful thought, and the 
interpretation and discussion of results should be intimately 
tied to the chosen definition. 

An Illustration 

In this section we provide an empirical example using 
the ECLS-K to demonstrate the impact of the rural definition 
on parameter estimates and substantive interpretations. The 
ECLS-K is a longitudinal study that followed a nationally 
representative sample of children from kindergarten 
entry (1998-1999 school year) through eighth grade. The 
restricted-use license provides access to the census tract 
and county FIPS codes for each participating child’s school, 
which allowed us to merge existing rural definition codes 
with the dataset. 

It is important to stress that this example is provided 
solely for demonstration purposes. The results from our 
analyses should not be interpreted in any real manner. We 
did not apply sampling weights or adjust for additional 
layers of design complexity (e.g., stratification, additional 
levels of clustering) as is normally required when analyzing 
data from the ECLS-K. Our goal is simply to highlight 
differences that may result when choosing among different 
modeling approaches and different rural definitions. 

Method 

Using data from the spring third grade wave, we 
compared students’ science scores across urban and rural 
locations as defined by the school-level Metro-Centric 
Locale Codes 5 (drawn from the 1999-2000 Private School 
Universe Survey [PSS] and 2000-2001 Common Core of 
Data [CCD]; NCES, 2003), census-tract-level RUCAs 
(Version 2.0, based on data from the 2000 U.S. Census; 
WWAMI RHRC, n.d.-b), and 2003 county-level OMB 
designations (based on the 2000 OMB Standards; OMB, 
2000). As a whole, these definitions map on most directly 
to the demographic- and spatially-based theories of rural. 
All three definitions are derived directly or indirectly from 
population size and density, and relation to a CBSA. The 

5 The ECLS-K dataset does not include the newer 
Urban-Centric Locale Codes. 


Metro-Centric Locale Codes additionally distinguish among 
types of incorporated places (e.g., central city vs. town) and 
census-designated places. The OMB and RUCA definitions 
additionally take commuting patterns into account. The 
RUCA definition goes further by distinguishing areas based 
on their primary and secondary flows. 6 In their full form, 
the Metro-Centric Locale Codes and RUCAs provide a 
much finer-grained measure of rurality than the OMB 
classification. However, for practical reasons, researchers 
often combine the codes into considerably fewer categories. 
Consequently, differences among definitions become less 
pronounced. For this illustration we used a two-category 
classification scheme (urban vs. rural) that was obtained 
from the Rural and Low-Income School Program for the 
Metro-Centric Locale Codes (codes 1-5 were designated 
as urban; all other codes were designated as rural; U.S. 
Department of Education, Office of Communications and 
Outreach, 2012), the WWAMI RHRC website for the 
RUCAs (codes 1.0, 1.1, 2.0,2.1, 3.0, 4.1, 5.1, 7.1, 8.1, 
and 10.1 were designated as urban; n.d.-c), and based on 
common practice by many federal programs for the OMB 
classification (metropolitan counties were designated as 
urban; Coburn et ah, 2007). Note that the two-category 
classification scheme for the RUCAs is an approximation 
of the two-category classification scheme for the OMB 
definition, but at the census tract level (WWAMI RHRC, 
n.d.-c). 

Analyses were performed in M plus Version 6.1 
(Muthen & Muthen, 1998-2010). Syntax is available from 
the first author upon request. We first analyzed a set of 
single-level models that ignored the dependency among 
observations. We then estimated a set of multilevel models 
that accounted for the nesting of children within geographic 
units (where the geographic unit varied by definition). 
Finally, we estimated the same multilevel models but 
additionally controlled for child- and location-level SES. 
Analyses were limited to cases with complete data on all the 
variables of interest, which resulted in an effective sample 
size of 12,270 children, 2,440 schools, 2,240 census tracts, 
and 290 counties. 7 

Results 

Approximately 14% of sample schools were classified 
as rural based on the Metro-Centric Locale Codes, 11% 
of sample census tracts were classified as rural based on 
the RUCAs, and 19% of sample counties were classified 
as rural based on the OMB definition. Although there was 

6 See the respective sources for a more complete 
description of each definition’s codes. 

’Sample size numbers have been rounded to the nearest 
10 per Institute of Education Sciences’ restricted-use data 
reporting guidelines. 
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considerable overlap among definitions, cross-classification 
did occur. Four percent of rural schools were located in 
urban census tracts, and 3% of urban schools were located 
in rural census tracts. In addition, 4% of rural schools were 
located in urban counties, and 5% of urban schools were 
located in rural census tracts. Finally, 2% of rural census 
tracts were located in urban counties, and 4% of urban 
census tracts were located in rural counties. The estimated 
ICCs indicated that 33% of the variance in science scores 
existed at the school level or higher, 32% existed at the 
census tract level or higher, and 15% existed at the county 
level or higher. 

Table 2 provides the model-estimated means and 
mean differences in third grade science scores across rural 
definitions. The results corresponding to the first set of 
analyses are listed under “Single-Level, SES Omitted.” 
There was a significant mean difference in science scores 
between urban and rural counties based on the OMB 
classification (B = -0.07, SE = 0.02, p < .001) such that 
urban counties tended to outperform rural counties. There 
was no significant difference in science scores between 
urban and rural schools based on the Metro-Centric Locale 
Codes (B = 0.02, SE = 0.02, p = .308) or between urban 

Table 2 


and rural census tracts based on the RUCAs (B = -0.03, 
SE = 0.02, p = .150). The results under “Multilevel, SES 
Omitted” correspond to the second set of analyses. Urban 
counties significantly outperformed rural counties based 
on the OMB classification (B = -0.13, SE = 0.06, p = 
.031). In contrast, rural schools significantly outperformed 
urban schools based on the Metro-Centric Locale Codes 
(B = 0.10, SE = 0.03, p = .003). There was no significant 
difference between urban and rural census tracts based on 
the RUCAs (B = 0.05, SE = 0.04, p = .185). The results 
under “Multilevel, SES as Covariate” correspond to the final 
set of analyses. Upon controlling for child- and location- 
level SES, all three definitions indicated that rural locations 
significantly outperformed urban locations (B = 0.20, SE 
= 0.02, p < .001 for the Metro-Centric Locale model; B = 
0.19, SE = 0.03, p < .001 for the RUCA model; and B = 
0.12, SE = 0.04,p = .004 for the OMB model). 

Discussion 

As evidenced by our demonstration, the statistical 
model (single-level vs. multilevel model), rural definition 
(the indicators used to define rural, the geographic unit to 


Estimated Means and Mean Differences in 3 lcl Grade Science Scores for "Rural ” and “ Non-Rural" Locations 


Single-Level, Multilevel, Multilevel, 

SES Omitted SES Omitted SES as Covariate 



B 

SE 

P 

B 

SE 

P 

B 

SE 

P 

Metro-Centric Locale Codes 

Mean Urban Science Score 

-0.33 

0.01 

< .001 

-0.40 

0.02 

< .001 

-0.38 

0.01 

< .001 

Mean Rural Science Score 

-0.31 

0.02 

< .001 

-0.30 

0.03 

< .001 

-0.18 

0.02 

< .001 

Mean Difference 

0.02 

0.02 

.308 

0.10 

0.03 

.003 

0.20 

0.02 

< .001 

RUCAs 

Mean Urban Science Score 

-0.32 

0.01 

< .001 

-0.40 

0.02 

< .001 

-0.37 

0.01 

< .001 

Mean Rural Science Score 

-0.35 

0.02 

< .001 

-0.34 

0.03 

< .001 

-0.19 

0.02 

< .001 

Mean Difference 

-0.03 

0.02 

.150 

0.05 

0.04 

.185 

0.19 

0.03 

< .001 

OMB Classification 

Mean Urban Science Score 

-0.31 

0.01 

< .001 

-0.26 

0.03 

< .001 

-0.33 

0.02 

< .001 

Mean Rural Science Score 

-0.38 

0.02 

< .001 

-0.38 

0.05 

< .001 

-0.21 

0.04 

< .001 

Mean Difference 

-0.07 

0.02 

< .001 

-0.13 

0.06 

.031 

0.12 

0.04 

.004 


Note. B = unstandardized estimate of mean or mean difference. The third set of analyses controlled for child-level and location- 
level (school-level, census tract-level, or county-level) SES, where SES was grand-mean centered. 
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which the definition was applied, and the way in which 
the codes were combined), 8 and exclusion/inclusion 
of covariates greatly impacts parameter estimates and 
substantive conclusions. If this study was real and we 
had used a single-level or multilevel model based on the 
OMB definition without consideration of SES, we would 
have concluded that urban locations tend to have higher 
science test scores than rural locations. If we had instead 
used a single-level model without consideration of SES 
based on the Metro-Centric Locale Codes, or a single-level 
or multilevel model without consideration of SES based on 
the RUCAs, we would have concluded that urban and rural 
locations tend to perform equally well. Finally, if we had 
used a multilevel model without consideration of SES based 
on the Metro-Centric Locale Codes, or if we had taken into 
account SES and used a multilevel model based on any one 
of the rural definitions, we would have concluded that rural 
locations tend to have higher science test scores than urban 
locations. 

Rather than try to provide a substantive interpretation 
of these results, we use this exercise instead to comment on 
the appropriateness of the various definitions and analytic 
decisions given the context of our demonstration. With 
respect to operationally defining rural, the Metro-Centric 
Locale and RUCA definitions are preferred over the OMB 
definition, as the OMB definition was not designed to make 
inferences about rural areas (OMB, 2013). In addition, the 
OMB definition is applied at the county level, and counties 
tend to be much more heterogeneous than schools and 
census tracts. Choosing between the Metro-Centric Locale 
and RUCA definitions is less clear and would require careful 
theoretical consideration by subject-matter researchers. 
Of course, a different definition altogether may be more 
appropriate given an alternative theoretical perspective. With 
respect to the various modeling approaches, the multilevel 
models are more appropriate than the single-level models 
because they account for similarities among children who 
are nested within the same geographic location. Likewise, 
including relevant covariates is preferred over examining 
the rural effect in isolation, as covariates help to disentangle 
seemingly rural differences from differences that are 
actually explained by other variables such as SES. 

Conclusion 

8 Our comparison of the three rural definitions is 
confounded in the sense that we do not know whether the 
observed differences are due to differences in the indicators 
used to define rural, the geographic unit to which the 
definition was applied, or the way in which the codes were 
combined. Rather than using existing definitions, researchers 
could create new definitions that are more comparable by 
fully crossing the levels of these three factors. 


Due to the many theoretical perspectives and 
operational definitions of rural, rural researchers must 
shoulder extra responsibility to ensure that their work is 
maximally informative and easily replicable. To this end, 
we noted throughout our discussion that researchers should 
familiarize themselves with the most common definitions 
available today, and choose (or develop) a definition 
that, above all else, reflects their theoretical perspective. 
Supplemental considerations, such as the feasibility of a 
definition given the structure of the data and the intended 
analyses, are also important. Since rural research typically 
involves clustered data (e.g., children within classrooms, 
schools within counties), we described multilevel modeling 
as a potentially useful means for analyzing rural data. 
We also encouraged the use of covariates to account for 
additional variance in the model and rule out alternative 
explanations to rural phenomena. Finally, we reminded 
researchers to fully communicate the rural nature of their 
study, from their theoretical perspective and choice of an 
operational definition to situating their research findings 
within the context of other rural definitions. This approach 
provides the necessary detail to spur replication and 
encourage inclusion in meta-analytic work. 

Researchers also need to be cognizant of how modeling 
decisions and the choice of an operational definition 
of rural affect research outcomes and policy decisions. 
Using the ECLS-K dataset, we demonstrated the extent to 
which parameter estimates and substantive interpretations 
can differ across statistical models, rural definitions, and 
exclusion/inclusion of covariates. Inappropriate selection, 
analysis, and/or communication of the rural definition may 
result in misinformed conclusions about rural phenomena, 
which in turn may result in misinformed policy and program 
eligibility decisions. 

It is our hope that this article has provided researchers, 
both new and seasoned, with firm guidance in several 
critical aspects of rural research. We believe that more 
consistent and informed consideration of the presented 
guidelines will enhance and clarify the quantitative 
literature on rural education. We strongly encourage 
qualitative and mixed-methods researchers to continue our 
conversation on rural in order to provide a more complete 
set of guidelines. In addition, we encourage conversations 
such as those of Howley et al. (2005) that extend beyond a 
particular method. A blending, or at least a comparison, of 
perspectives is desired. Finally, we certainly acknowledge 
that rural education research is complex and each study is 
unique. There is no simple solution to the issues presented 
in this article. Informed and deliberate decision-making 
should always trump strict adherence to guidelines. 
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